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ABSTRACT 

Responses to a followup request for survey completion 
provided information about reasons for nonresponse by the target audience. 

Few studies of nonresponse have relied on reasons given by those who did not 
choose to respond; although not intentionally gathered, the data provide 
insights into nonresponse. An electronic mail survey was sent to 452 college 
deans, and 232 eventually responded. In all, there were four reminders, two 
by postcard and two by e-mail. Fifty-five deans replied to the e-mail 
followups and offered various reasons for not completing the survey. These 
replies were analyzed and grouped into five categories. Some simply refused 
participation, usually in a courteous and succinct way. The second category 
contained responses related to investing time to participate, and the third 
category included responses from persons who no longer served as deans and 
thus declined to respond. The fourth group declined to complete the survey 
because they only answer surveys prepared as part of the business of national 
organizations or surveys not found at Web sites. The final group declined to 
answer because they thought the survey was poorly designed or did not capture 
germane issues of leadership. Although it is not clear that these "repliers" 
were representative of the nonresponders overall, their e-mail comments 
provide some insight into nonresponse to Web surveys. (Contains 24 
references.) (SLD) 
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WEB SURVEYS AS A SOURCE OF NONRESPONSE EXPLICATION* 

Introduction 

In 2001, the Higher Education Leadership Instrument (HELI), was developed to assess 
the construct of higher education leadership; as part of its design, the HELI was administered 
over the internet (Montez, 2002). In order to increase response, followup messages were sent to 
nonrespondents by post and electronic mail; over the course of a four-month data collection 
period, four email reminders were sent. Of the 452 sample members, 55 (12.2%) replied to these 
reminders, declining participation and, in most cases, explaining their reasons for doing so. 
Examined through the lens of response behavior theory, this paper reports the unintended 
consequence of nonresponse explication from true nonresponders. 

Nonresponse 

A viable source of error in surveys, nonresponse occurs when a respondent is unable to 
gather information from a sizable portion of the sample, the composition of which may have 
some characteristic(s) that differ from the respondents (Dillman, 2000; Salant & Dillman, 1994; 
Yu & Cooper, 1983). In other words, nonresponse poses a threat to the survey's external 
validity— the ability to generalize that the nonresponders are representative of the study 
population under consideration (Viswesvaran, Barrick & Ones, 1993). 

Understanding antipathy to surveys is essential to optimize use of this medium for 
research. It is imperative for the following reasons: (a) the world-wide web presents an 
unprecedented opportunity to study groups of people with a questionnaire of this sort and, as a 
result, there has been a proliferation of online surveys (Schmidt, 1997; Supovitz, 1999); 

'In this paper, the term “nonresponse” refers to “unit nonresponse,” or the loss of one 
person’s response to the entire survey, instead of “item nonresponse,” which indicates the 
person’s failure to answer one or some of the items in the survey. 
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(b) internet-based survey research has often resulted in inadequate levels of participation (Pitkow 
& Recker, 1995);^ and (c) there is a stated need that online research demands data collection and 
analysis methods that are medium-specific (Witmer, Colman & Katzman, 1999). 

Still, there is little explanation of nonresponse itself (Bosnjak & Tuten, 2001). Indeed, 
what nonresponders “look like” and why they become nonrespondents is a rarely known bias 
source in survey investigations (Lubin, Levitt & Zuckerman, 1962). Given the difficulty in 
assessing the true nature of nonresponse, more has been written about techniques for increasing 
response rates and/or for estimating or correcting for nonresponse bias post survey (Bosnjak & 
Tuten, 2001; Dillman, 2000; Groves, Cialdini & Cooper, 1992). 

Where research efforts have concentrated on explaining nonresponse, most study designs 
have focused on: 

1 . Variations in response behaviors fi'om surveys about nonresponse (“surveys on 
surveys”), where surveys asked why persons do not respond to surveys and the 
respondents’ input was used as the basis for nonresponse (Bosnjak & Tuten, 

2001; Goyder, 1987; Sjoberg, 1954); 

2. “Record-linking,” where sociodemographic information on sample individuals 
was obtained through internal and external sources, then compared for 
respondents and nonrespondents (Ellis et al., 1970; Robins, 1963); or 



^Indeed, although a 50% response is typically considered minimally adequate for much 
traditional survey research (Babbie, 1998), response rates around 20% are not uncommon for 
unsolicited surveys and response rates to online surveys may be 10% or lower (Patrick, Black & 
Whalen, 1995). 
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3. Panel studies, where repeated enumerations of a survey were given to the sample 
and responses and sociodemographic characteristics of those who fell from the 
panel were then compared to those of the sample who remained (Goyder, 1987). 

In other words, most data that exist about nonresponse have been based on responses 
given in surveys and the conclusions derived therefrom that were imputed, based on presumed 
sample characteristics, to nonrespondents. There is little that reflects the impressions of true 
nonresponders (Bosnjak & Tuten, 2001). 

In their work with internet surveys, Bosnjak & Tuten (2001) devised a methodology that 
allows for understanding the underlying psychology of response. While the method they 
employed tracks the process and progress of sample members in accessing a survey website, the 
end product does not distinguish whether nonresponse was based on refusal or on inability to 
access the website. In other words, though their study profiled respondents’ activity (or the lack 
thereof) at the website, they were unable to explain why the choice to not respond was made 
(Bosnjak & Tuten, 2001). 

This problem was also addressed by Goyder (1987): 

Asking nonrespondents to at least articulate motives for refusal ... is worth 
considering in combination with other forms of evidence, but taken in isolation 
remains severely limited. That those hostile to surveying will deliver well- 
considered attitudinal reports to the fieldworker surely is debatable, p. 21-22 
Response Behavior Theory 

Early response behavior theories focused primarily on sociodemographic categorization 
of respondents and the deficiencies of those who did not respond. Indeed, nonrespondents were 
often regarded as deviants, members of a “socially isolated underclass” (Goyder, 1987, p. 15). 
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Others referred to them as “bad buys,” where responders were “good guys” (Kirchner & 
Mousley, 1963). In these earlier studies there was a presumption that nonresponders were less 
educated, had lower occupational status and had lower incomes (Goyder, 1987). 

Obviously, these earlier conclusions were drawn from experimental studies that focused 
more on mechanics of the study than the subjects themselves, resulting in a lack of “natural 
variance” (variation in response behavior attributable to humans). Best efforts to describe 
nonresponse were couched in terms of sample members' refusal to respond, their inability to 
respond, or their inaccessibility to the researcher (Williams, 1978). 

Rather than categorizing nonrespondents according to sociodemographic status, Goyder 
(1987) argued that response (or nonresponse, as the case may be) behavior is a “rational decision 
making [process] in which the respondent balances costs against reward for action.” (p. 163) 

The notion of response behavior as social exchange (i.e., there is a promise of a reward in 
exchange for little expenditure of cost and the expectation that doing something will outweigh 
the costs) was most popularly advanced by Dillman (2000). The practical effect of this theory is 
that respondents answer surveys if some “thing” (personalization of contact, association with an 
authority, financial incentive, aesthetically pleasing document) is given in return. 

Goyder’s concept of response behavior is that it possesses both a rational (conscious, 
volitional) as well as a deterministic (nonresponse is a reaction to the social environment) 
component. It is this synthesis of behaviorism and voluntarism, he argues, that enriches 
understanding survey response behavior (1987). 

Goyder asserted that respondents choose to answer surveys and nonrespondents invoke 
the same decision-making consciousness; in other words, they simply make the decision not to 
respond: 
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Nonrespondents assess the costs and benefits of participation in a survey 
differently than respondents do, these calculations resting on matters as deep 
within the practical consciousness as an unarticulated sense that social research 
seldom serves the collective interests of their class or generation, and as near the 
top of the discursive consciousness as the resistance of professors indifferent to 
computers to responding to survey questions on that topic; . . . and as profound as 
a careful judgement that surveys erode the dignity of the individual. Part of the 
explanation for response behaviour rests on people’s assessments about the merits 
and demerits of surveys in general, but as much, perhaps more, is unique to the 
particulars of each survey, (p. 187) 

Groves, Cialdini & Couper (1992) articulated a model that incorporates influential and 
psychological response factors, a model that lends a framework to Goyder’s nonresponse 
multidimensionality. The influence factors that affect response are societal characteristics (social 
responsibility, perceived legitimacy of social institutions), survey design attributes (contacts, 
length and topic of survey), sociodemographic (age, gender, income, health status), and the 
resulting interaction of these influences (Groves et al., 1992). 

The psychological factors explain a respondent’s decision to comply with survey 
requests. The survey’s attractiveness (the interest value and personal relevance of the survey to 
the respondent as well as the cost in time, energy, and resources to perform it) is a common 
deciding factor for response participation (Groves et al., 1992). In addition, Cialdini (1988) 
outlined six principles that explain respondents’ decision to comply (or not) with a survey 
request: 
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1 . Reciprocation— where the respondent should be willing to comply with a request 
to the extent that compliance constitutes repayment of a perceived gift, favor, or 
concession; 

2. Consistency— after committing to a position, the respondent should be more 
willing to comply with requests for behaviors that are consistent with that 
position; 

3. Social validation — one should be willing to comply to the degree that similar 
others comply with the request; 

4. Authority — the respondent would be more willing to yield to the requests of 
someone perceived to be a legitimate authority; 

5. Scarcity — one should be more willing to comply with requests to secure 
opportunities that are scarce; and 

6. Liking — the respondent should be more willing to comply with the requests of 
liked others (Groves et al., 1992). 

HELI 

The Higher Education Leadership Instrument (HELI) was developed to assess five 
dimensions of the construct of higher education leadership, and the author’s dissertation 
described the instrument’s developmental and pilot processes (Montez, 2002). Considerations 
that informed the decision to administer this study over the internet were the growing popularity 
of this medium for conducting research, the proliferation of this technology in higher education, 
and the potential to more easily access the target population (Dillman, 2000; National 
Telecommunication & Information Administration, 1999). In addition, because web surveys are 
dynamic interactive forms, their usage has been shown to enhance respondent motivation 
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(Dillman, 2000; Schmidt, 1997). Cost was also a consideration; administering a web survey has 
been shown to be low compared to traditional pencil and paper survey techniques (Dillman, 
2000; Pitkow & Recker, 1995; Schmidt, 1997). 

A domain was acquired for this survey and a website was created and maintained by a 
third-party internet service provider (ISP). In order to prompt respondents to provide thoughtful 
and accurate responses, design features included program protocols written to allow 
automatic feedback in the form of repeating scales, generate a status report on the number of 
questions remaining for answer, and issue alerts of incomplete or unacceptable responses (e.g., 
numbers instead of words) (Schmidt, 1997). Program subroutines were also written to prevent 
multiple submissions, preserve data integrity, and protect respondents’ anonymity. Access was 
only allowed by using a unique PIN which was assigned to each respondent. Respondents were 
able to save their responses for later return and revision; however, once the instrument was 
completed and submitted, further access was denied. 

The HELI was constructed in accord with Dillman’s (2000) design principles for internet 
surveys. The instrument contained 64 scales, a series of demographic information requests, and 
several open-ended questions about respondents’ impressions of the instrument and/or its 
administration. The survey instrument was prepared using Microsoft Word and posted to the 
website as an HTML document. The introductory page contained a welcome message and a fill- 
in box into which the PIN was typed. The author’s email address was provided on this page to 
facilitate any questions or concerns that respondents had before proceeding to the instrument. A 
Microsoft Excel spreadsheet containing the respondents’ names, email addresses, PINs, and 
other identifying information was prepared and also served as the data capture document. On 
verification of the PIN with the respondent’s name in the spreadsheet, the data entered at the 
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HELI website were written to a Microsoft Standard Query Language (SQL) database maintained 
on the ISP’s server. Javascript was used to validate that the required fields contained data. 

Open Database Commands (ODBC) were used to transform the SQL data into Excel format and 
the data were posted to the spreadsheet. Only the researcher was allowed to manage access to 
the website and to download the Excel spreadsheet containing the most current collected 
responses. 

A sample of 452 deans of colleges of agriculture, arts & sciences, business, education, 
engineering, and nursing was randomly drawn from the Higher Education Directory, 2001 ed. 
(Rodenhouse, 2000). Notice of the study was first made by mailing a letter to each respondent, 
advising of his/her selection for participation and providing background information on the 
study. The letter also contained a 3"x5" card that contained the URL (“uniform resource 
locator,” or address) of the HELI website, the available time frame for completing the web 
survey (July through September, 2001), the personal identification number (PIN) assigned to the 
respondent, and an email address for the researcher. 

The initial letter and PIN were mailed to sample members in July, 2001 and the HELI 
website went “live” with data collection beginning on August 1, 2001. Given the increasing 
trend of email use for people in their jobs, it was decided to use this medium for several follow- 
up contacts (Dillman, 2000; Salant & Dillman, 1994). A total of six reminders were made, two 
by postal mail, and four by email. The survey protocol was modified midway through data 
collection to offer an optional paper version of the HELI. The paper version closely matched the 
web version, including paper color and font styles, with wording changes made only to delete 
references to the internet. All sample members who had yet to respond to the HELI were 
advised of this modification. Data collection was terminated in November, 2001. 
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Findings 

In all, 232 responses were received from the deans in the sample (5 were returned for 
lack of a good postal or email address), for a response rate of 51%. Of this number, 16 answered 
the paper version of the HELI (6.9%). In the process of trying to achieve a higher response rate 
in this study using email follow-up contacts, an interesting phenomenon emerged. Of the 21 8 
deans who did not respond to the HELI, 55 sent replies to the email followups and offered 
various reasons for not doing so. (It is uncertain how many of the 55 nonrespondents actually 
accessed the survey items before declining to answer, but none used the open-ended questions at 
the end of the survey to convey their thoughts.) 

The 55 nonrespondents’ communications were compiled and analyzed, following the 
convention on data management and analysis described by Huberman and Miles (1994). The 
data were coded for similarity then condensed. A text matrix of similar themes was then 
constructed, allowing a display of the items for analysis; patterns and themes and how they 
clustered were noted and are discussed below. 

Five categories derived from these responses. In the first category, the nonrespondent 
refused participation without citing any reason therefor. Most responses were courteous and 
succinctly worded: 

I am sorry that I am unable to help you 
I do not plan to take part in this survey 
I decline to participate 

The second category included responses related to investing time to participate. 

Examples given were: 

Dean [Doe’s] schedule will not allow her to participate. 
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I already tried to respond, so I am not inclined to invest more at this time. 

I am sorry, but I do not have the time to participate in this study. 

Some respondents stated that they could not answer because they were far behind in their work, 
and one stated an overcommitment to other projects did not allow for taking time to complete the 
survey. One dean noted that he had attempted to complete the survey on the web, but was unable 
to, and thus declined to attempt it again for lack of time. (However, this dean did express an 
interest in the instrument and “suggested to some of our doctoral directors that they use the same 
technique.”) Several provided additional reasons for not finding the time to complete the survey, 
such as having to complete a large grant proposal, being trapped in New York during the week of 
September 11, 2001, and covering the responsibilities of an associate dean who had just died. 

One dean noted there was never a convenient time to complete the survey, as “dozens” were 
filled out each month. 

The third category included responses fi-om persons who no longer served in the position 
as dean and, as such, declined to respond. However, almost all of these deans provided the 
names and email addresses of their successors. 

The fourth group of persons declined to respond because they “only answer surveys that 
are prepared as part of the business of national organizations, such as AACTE” or those that are 
"not found at websites." 

The last group of nonrespondents declined to answer because they felt the survey was 
poorly designed, or did not capture germane issues of leadership. One person stated the items 
were not “sufficiently refined to enable a knowledgeable person to reply conscientiously, without 
thinking to himself or herself, ‘This question offers insufficient options or suggests by its 
language ideas that are not true, and so I cannot reliably answer it.’” Another person stated 
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“Expecting recipients to go to websites and provide answers to lengthy forms is really not a very 
realistic, or collegial, manner of collecting information.” 

Discussion 

The five categories of nonresponse reasons provided insight into the rationale employed 
by some deans for not responding to the HELI, and confirmed several of the factors described in 
Groves et al.’s (1992) model. The first group simply refused to participate, citing no reason for 
doing so. 

Though lack of time and being too busy were cited the most as impediments by the 
second group, these deans conscientiously took the time to explain why they were unable or 
unwilling to respond. The nonrespondents in this category exhibited noncompliance with the 
request to participate because of the cost in time, energy, and resources to them (Groves et al., 
1992). That they provided a reason for their nonresponse on this basis underscores Goyder’s 
(1987) statement that “ . . . the ‘irrationality’ [that] survey researchers might feel tempted to 
assign to those who refuse a siuwey is fully rational from the actor’s [nonrespondent’s] 
perspective.” (p. 186) Here, the respondents rationalized that the expense of time to complete 
the survey was outweighed by the reason they gave. 

In the third category, some deans who had earlier stepped down from their positions 
replied by sending their successors’ addresses, allowing the new dean to answer in his/her stead. 
Though invited to participate, most of these former deans declined to answer, stating “I am no 
longer a dean.” The responses in this category fit the “social validation” factor, where 
compliance occurs because similar others (i.e., deans) do so. The declining deans evaluated their 
position (by virtue of retirement or “stepping down” from the deanship) relative to others and the 
lack of similarity apparently drove their decision to not participate. 
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One way of handling the overwhelming number of surveys that deans receive is to be 
selective about those that they do care to answer. The fourth category of nonrespondents 
grouped deans who choose to answer only surveys from specific organizations or associations 
and decline response to the rest. Clearly, this reasoning supports Groves et al.’s (1992) 
“authority” factor, which states that survey compliance occurs if it comes from a properly 
constituted, legitimate authority. In other words, a respondent would rationalize that a study’s 
affiliation creates sufficient trust that the study has some legitimacy. The HELI had no sponsor 
and, thus, the sample members in this group obviously found no reason to legitimize it. 

The last category of nonrespondents captured comments from deans who consciously 
refused to respond, stated their decision, then explained their rationale for doing so. 

Examination of their messages clearly reflects a greater thought process beyond simple 
compliance or declining to respond; indeed, it belies the “ignorance and incapacity to reflect” on 
the part of nonrespondents as earlier studies had postulated (Goyder, 1987, p. 186). For 
example, the respondent who felt the HELI did not capture germane issues of leadership went on 
to explain what those issues were and why she felt they were germane. Her email printed out to 
almost two single-spaced pages— the time to drum out this response surely equaled or exceeded 
that for completing the survey. Another respondent followed up his initial email with another in 
which he articulated further thoughts about the inappropriateness of web surveys in a construct 
validation study. 

The respondents’ insights were very keen and, surely, would not have found place in any 
of the response categories provided in the web survey. This last category of reasons is perhaps 
the strongest evidence of nonresponders’ volitional, rational mechanism at work. 
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Study Limitations, Implications, and Recommendations 

In this study, email followups were used to remind sample members to complete the 
HELL Rather than doing so, nonrespondents used this means to explain their nonresponse, 
perhaps because it more quickly captured and conveyed their thoughts and impressions. It seems 
unlikely that a post-mailed reminder would have elicited the same response. 

And, though some light was shed upon the issue of nonresponse, it is still uncertain that 
these nonresponders were indeed representative of the nonresponding group. As noted, only 
12.1% of the entire sample declined response while explaining their reason for doing so; the 
remaining 37% were silent. 

Another limitation to this study is that the sample comprised a homogeneous population- 
academic deans drawn from one sample frame. The fact of their advanced education and 
employment-related accessibility to email and the internet may actually have contributed to the 
number of responding nonrespondents. 

In spite of this contribution to our knowledge of nonresponse, it must be remembered that 
the purpose of surveys is to elicit the greatest number of responses from sample members. Thus, 
identifying the reasons for nonresponse must be considered secondary to, or a byproduct of, the 
survey process, and care must be taken to avoid encouraging nonresponse in surveys that use a 
parallel response mechanism such as email. 

Still, using email as a communication mechanism during the web survey administration 
in this study offered a prime opportunity to explore the issue of nonresponse. Though the HELI 
continues to be refined, it is expected that the protocol for its administration will not change; 
followup email reminders will continue to be used and, in all likelihood, further data will be 
added to this small but significant nonresponse database. 
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Affording sample members this low-cost means to say “something” about why they 
chose not to respond undoubtedly will contribute to further understanding of this phenomenon 
which in turn will inform future web survey study design. As we continue to learn more about 
true nonresponse, our knowledge will inform improved and changed ways to optimize response 
in surveys. Groves et al.’s (1992) model provides an excellent framework to guide such change. 
As nonresponse manifests itself through email replies outside the web survey, comparing them 
against the model will clarify the work to enhance response in that area. For example, where 
sample members decline to participate because the instrument lacks authority, the researchers 
may seek sponsorship from an organization to provide legitimacy in future study administration. 
Similarly, sample members' refusal to participate for lack of time may inform a revision in 
design to allow for longer periods of data collection, improved means to suspend survey 
completion without data loss, or the production of an optional paper version of the instrument so 
the responder can choose the time or location for its completion. 

Beyond surveys, the data gathered in this study argue for augmenting the research design 
with qualitative methods, such as interviews or written narratives, to account for the responses of 
those sample members who wish to contribute their knowledge but through a different vehicle 
than a survey. Emplo}dng a variety of methods in research more completely defines the 
construct under study (Babbie, 1998). As we continue to fine-tune web surveys, it might be 
worthwhile to examine possibilities for conducting qualitative research over the internet, either 
independent of, or simultaneously with, the web survey. Clearly, with well developed controls 
built into design, research over the internet has yet to be exploited. 
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Conclusion 

As the internet continues to gain popularity as a preferred medium for survey 
administration, research that can identify the sources of bias to reduce nonresponse will help to 
validate its use. This unintended outcome of the HELI study has given us a glimpse of true 
nonresponse and it is hoped that its small contribution adds to that research. 
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